[BugFix] Seperate prometheus multiproc dir for single-server multi-dp services#8059
[BugFix] Seperate prometheus multiproc dir for single-server multi-dp services#8059liyonghua0910 wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8059 +/- ##
==========================================
Coverage ? 67.22%
==========================================
Files ? 475
Lines ? 66884
Branches ? 10317
==========================================
Hits ? 44961
Misses ? 19051
Partials ? 2872
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
liyonghua0910
force-pushed
the
develop+20260616_fix_dp_metrics
branch
from
d1ec8f0 to
15b4d28
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
CI报告基于以下代码生成（30分钟更新一次）: 1 Required任务 : 3/10 通过
2 失败详情🔴 Approval — 需要 Approval（置信度: 高）该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。 修复建议：请通过人工审批。 |
liyonghua0910
force-pushed
the
develop+20260616_fix_dp_metrics
branch
from
15b4d28 to
13e5962
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
liyonghua0910
force-pushed
the
develop+20260616_fix_dp_metrics
branch
from
13e5962 to
9a08ce4
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-06-22 11:03:00
📋 Review 摘要
PR 概述：为多 DP 服务拆分 Prometheus multiprocess 目录，避免多个 DP 共享同一指标目录。
变更范围：fastdeploy/metrics、fastdeploy/engine、OpenAI multi API server 与相关测试。
影响面 Tag：[Engine] [APIServer]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🔴 Bug | fastdeploy/metrics/prometheus_multiprocess_setup.py:47 |
用户指定的 multiprocess 目录被静默复用，旧 .db 会继续参与采集并污染 DP0 指标 |
历史 Findings 修复情况
| Finding | 问题 | 状态 |
|---|---|---|
| F1 | engine.py 切到 dp0 后 /metrics 仍只采集当前环境变量目录 |
|
| F2 | common_engine.py 启动路径也切到 dp0，但 metrics HTTP 入口未支持多个 DP 子目录 |
📝 PR 规范检查
PR 标题 tag 符合规范，但描述各章节仍是模板占位内容，建议替换为下面可复制版本。
标题建议（可直接复制）：
[BugFix] Separate prometheus multiproc dir for single-server multi-dp services
PR 描述建议（点击展开，可直接复制）
## Motivation
修复单 API server 多 DP 部署中 Prometheus multiprocess 目录共享导致 Counter/Histogram 指标容易混淆的问题。
## Modifications
- `fastdeploy/metrics/prometheus_multiprocess_setup.py`: 新增 `setup_dp_prometheus_dir`，按 DP id 创建独立的 Prometheus multiprocess 目录。
- `fastdeploy/engine/engine.py`、`fastdeploy/engine/common_engine.py`: 启动 DP 子服务前切换到对应 DP 的 `PROMETHEUS_MULTIPROC_DIR`，子服务启动后切回 DP0 目录。
## Usage or Command
N/A
## Accuracy Tests
N/A（仅调整 metrics multiprocess 目录，不影响模型精度）
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
目录隔离方向合理，但当前实现仍没有补上单 server 多 DP 子目录的聚合采集，历史两个指标完整性问题仍存在；同时本次把用户指定 multiprocess 目录从 warning 改为静默复用，会让陈旧 .db 文件继续污染本轮 metrics。建议先修正目录生命周期和多 DP 采集语义后再合入。
|
|
||
| user_dir = os.environ["PROMETHEUS_MULTIPROC_DIR"] | ||
| _original_prom_dir = user_dir | ||
| os.makedirs(user_dir, exist_ok=True) |
There was a problem hiding this comment.
🔴 Bug 这里在用户传入的 PROMETHEUS_MULTIPROC_DIR 上直接 os.makedirs(..., exist_ok=True) 并继续复用目录，但没有清理已有 .db 文件。
get_filtered_metrics() 仍会通过 multiprocess.MultiProcessCollector 读取当前目录里的所有 .db，而后续 setup_dp_prometheus_dir(0, base_dir) 还会把 base 目录下的旧 .db 迁到 dp0/。固定复用同一个目录或服务异常退出后，上一轮进程的 counter/histogram 会被当成本轮 DP0 指标采集，PR 反而会静默产生不准确 metrics；旧代码至少会 warning 这个风险。
建议修复方式：在注册 metrics 前保证本轮使用的 base/dp* 目录没有旧 .db，例如为用户目录再创建带 UUID 的 run 子目录作为 base_dir，或显式删除 base 与对应 dp{i} 子目录中的 .db；如果不打算接管用户目录清理，则保留 warning/校验并拒绝非空目录。
3 participants
Motivation
Modifications
Usage or Command
Accuracy Tests
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.